The effect of discrete vs. continuous-valued ratings on reputation and ranking systems

When users rate objects, a sophisticated algorithm that takes into account ability or reputation may produce a fairer or more accurate aggregation of ratings than the straightforward arithmetic average. Recently a number of authors have proposed different co-determination algorithms where estimates of user and object reputation are refined iteratively together, permitting accurate measures of both to be derived directly from the rating data. However, simulations demonstrating these methods’ efficacy assumed a continuum of rating values, consistent with typical physical modelling practice, whereas in most actual rating systems only a limited range of discrete values (such as a 5-star system) is employed. We perform a comparative test of several co-determination algorithms with different scales of discrete ratings and show that this seemingly minor modification in fact has a significant impact on algorithms’ performance. Paradoxically, where rating resolution is low, increased noise in users’ ratings may even improve the overall performance of the system. Introduction. – With the growth of the internet and e-commerce [1], an increasing number of our social and commercial interactions are now one-shot exchanges with strangers identifiable only by easily-replaced pseudonyms [2]. Similarly, most items on sale from ecommerce websites must be purchased without an opportunity to try them first, creating an information asymmetry that encourages the provision of low-quality goods [3]. To offset this risk of fraud or deception, many online services implement reputation systems [4] that collect ratings and feedback from users so as to provide a measure of trustworthiness for goods or individuals. A key challenge is how to aggregate this feedback effectively given that not all ratings are equal. Some users’ judgement may be poor or malicious: for example, many eBay users forgo issuing negative feedback to cheaters because the mendaciously negative response will devastate their own carefully cultivated good reputation [5]. An effective reputation system thus needs to distinguish between good and bad raters and ratings. (a)E-mail: [email protected] (b)E-mail: [email protected]. Present address: CREATE-NET Via alla Cascata 56D, 38123 Povo di Trento, Italy. One approach to this has been the development of codetermination algorithms of reputation, where aggregate reputation (or quality) of rated objects is used to estimate a corresponding reputation (or ability) for the system’s users, and this latter measure is then used to re-weight the aggregation of ratings for objects [6–8]. By iterating this procedure over time, ratings from malicious or unskilled users can be weeded out, providing both a better estimation of object quality and an enhanced overall reputation-based ranking of objects. Simulations to evaluate the effectiveness of these methods followed typical modelling practices in physics and applied mathematics, assuming a continuum of rating values (reflecting what may be presumed to be fine-grained shades of opinion). However, a near-universal feature of real user feedback and rating systems is that they permit ratings to take only a limited range of discrete values— most commonly the 5-star system employed by Amazon, YouTube, etc. The influence of this constraint has never 1 We use ‘object’ simply as a generic term: the object of the rating. This might be an actual object, such as a book or CD, or it might be a person or organisation, such as an eBay auctioneer, a website, or an Amazon Marketplace seller.